Vehicle Occupant Monitoring Using Infrared Imaging

ABSTRACT

Methods and systems for monitoring vehicle occupants using infrared and other sensors are disclosed. The systems may use infrared sensor data to identify each vehicle occupant using biometric signatures (such as heartbeat or facial recognition). Vehicle occupant characteristics may be determined that include skeletal characteristics of the occupants. Further monitoring of the occupants using sensor data may occur until an abnormal situation is detected. Abnormal situations may include medical emergencies, driver impairment, security threats, or similar situations requiring correcting action. The system may then determine and implement an appropriate response to the abnormal situation. Such responses may include generating alerts, adjusting vehicle environmental controls, taking control of operation of the vehicle (such as an autonomous or semi-autonomous vehicle), or initiating wireless communication with an outside party, such as an emergency service.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims the benefit of, U.S.patent application Ser. No. 15/916,500, entitled “Vehicle OccupantMonitoring Using Infrared Imaging” and filed Mar. 9, 2018, which is acontinuation of U.S. patent application Ser. No. 15/248,073 (filed Aug.26, 2016 and now U.S. Pat. No. 9,988,055), which claims the benefit ofU.S. Provisional Application No. 62/213,256 (filed Sep. 2, 2015), theentireties of which are hereby incorporated herein by reference.

FIELD

The present disclosure generally relates to systems and methodsaddressed to vehicle occupant safety and health monitoring and responseusing infrared sensors to identify and assess occupant characteristicsor conditions.

BACKGROUND

Every year many vehicle accidents are caused by vehicle operatorsimpaired by drowsiness, illness, intoxication, rage, or distraction. Onecommon kind of impaired vehicle operation is drowsy driving. If thevehicle operator falls asleep for even a second while driving, theresults can be disastrous. Another common kind of impaired vehicleoperation is distracted driving. Modern motor vehicles come equippedwith any number of distractions including stereos, air-conditioners,navigation systems, etc. Furthermore, a vehicle operator may bedistracted by another passenger or by articles the vehicle operatorbrings into the vehicle (e.g., a mobile telephone, book, etc.). Yetanother common kind of impaired vehicle operation is agitated, anxious,or aggressive driving. Numerous incidents occurring during the course ofa trip may aggravate the vehicle operator, such as traffic jams, poordriving by other drivers, vehicle malfunctions, or inclement weatherconditions. Additionally, factors unrelated to the trip may distract oraggravate the vehicle operator, such as receipt of bad news, runningbehind schedule, passenger conduct, or any number of factors occurringprior to vehicle operation. These and other factors may impair theability of vehicle operators to operate vehicles safely.

Many modern vehicles are equipped with on-board computer systems thatcontrol some or all of the operational, environmental, and informationalfeatures of the vehicles. Additionally, many vehicle operators carrymobile devices (such as smartphones) with them while operating-vehicles.Such mobile devices often communicate with the vehicle in ways that mayallow the mobile devices to control portions of the vehicle features,such as external telephonic communication. Despite the availability ofcomputing resources within many modern vehicles, such resources are notused to detect and mitigate dangerous situations involving vehicleoperators or passengers. The methods and systems disclosed herein areaddressed to such detection and mitigation.

BRIEF SUMMARY

The present invention discloses a method, system, and computer-readablemedium storing instructions for determining and responding to abnormalor dangerous situations within a vehicle. The method, system, orcomputer-readable medium may operate to monitor and respond to abnormalsituations based upon sensor data received using one or more processors.The one or more processors may be disposed within an on-board computeror a mobile device associated with the vehicle or with a vehicleoccupant.

In accordance with the described embodiments, one or more processors maymonitor one or more vehicle occupants of a vehicle by receiving sensordata regarding the one or more vehicle occupants from one or moresensors disposed within the vehicle, determining one or more vehicleoccupant characteristics for at least one of the one or more vehicleoccupants based upon the received sensor data, determining whether anabnormal situation exists based upon the one or more determined vehicleoccupant characteristics, determining one or more responses to theabnormal situation based upon the one or more determined vehicleoccupant characteristics when an abnormal situation is determined toexist, and/or causing the one or more responses to the abnormalsituation to be implemented.

In some embodiments, the abnormal situation may relate to one or more ofthe following types of abnormal situations: a medical emergency, ahealth risk, an accident risk, an impairment of a vehicle occupant,and/or a security threat. Moreover, the one or more responses may bebased upon the determined type of the abnormal situation. In someembodiments, the at least one vehicle occupant may include a vehicleoperator controlling the vehicle. The determination of an abnormalsituation and/or the one or more responses may be based, at least inpart, upon whether the determined vehicle occupant characteristics areassociated with a vehicle operator or are associated with anothervehicle occupant, such as a passenger.

In further embodiments, the one or more processors may identify the atleast one of the one or more vehicle occupants based upon the one ormore determined vehicle occupant characteristics. This may includeidentifying the at least one of the one or more vehicle occupantsincludes comparing the one or more determined vehicle occupantcharacteristics with data regarding characteristics stored in a userprofile. When a user profile is used, determining whether the abnormalsituation exists may include determining whether the one or moredetermined vehicle occupant characteristics are beyond a baseline rangefor the vehicle occupant based upon the data regarding thecharacteristics stored in the user profile of the vehicle occupant.

In yet further embodiments, the one or more sensors may include one ormore infrared sensors disposed within the vehicle. Using the sensor data(which may include infrared and/or optical image data), the one or moreprocessors may determine vehicle occupant characteristics, which mayinclude one or more skeletal characteristics of the at least one of theone or more vehicle occupants. Such skeletal characteristics mayindicate the position of a plurality of segments of the vehicleoccupant's body, which plurality of segments may include the vehicleoccupant's head, torso, and/or at least a portion of a limb of thevehicle occupant.

In still further embodiments, the one or more responses may include oneor more of the following: controlling vehicle operation by an on-boardcomputer system, adjusting an environmental condition within thevehicle, communicating a message to an emergency response service,and/or terminating vehicle operation. Further responses may includecommunicating sensor data to one or more computing devices associatedwith emergency response personnel. Such communications may facilitatetheft recovery, medical care delivery, and/or security threat response.

The methods, systems, and computer-readable media may includeadditional, fewer, or alternate actions, including those discussedelsewhere herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Advantages will become more apparent to those skilled in the art fromthe following description of the preferred embodiments which have beenshown and described by way of illustration. As will be realized, thepresent embodiments may be capable of other and different embodiments,and their details are capable of modification in various respects.Accordingly, the drawings and description are to be regarded asillustrative in nature and not as restrictive.

The figures described below depict various aspects of the applications,methods, and systems disclosed herein. It should be understood that eachfigure depicts an embodiment of a particular aspect of the disclosedapplications, systems and methods, and that each of the figures isintended to accord with a possible embodiment thereof. Furthermore,wherever possible, the following description refers to the referencenumerals included in the following figures, in which features depictedin multiple figures are designated with consistent reference numerals.

FIG. 1 illustrates a block diagram of an exemplary vehicle occupantmonitoring system on which exemplary vehicle monitoring methods mayoperate in accordance with the described embodiments;

FIG. 2 illustrates a block diagram of an exemplary mobile device or anon-board computer for use in the vehicle occupant monitoring system;

FIG. 3 illustrates a flow diagram of an exemplary embodiment of a userprofile generation method for creating or updating a user profile;

FIG. 4 illustrates a flow diagram of an exemplary embodiment of amonitoring and response method that may be implemented using theexemplary vehicle occupant monitoring system;

FIG. 5 illustrates a flow diagram of an exemplary medical emergencymonitoring and response method that may be implemented by the vehicleoccupant monitoring system;

FIG. 6 illustrates a flow diagram of an exemplary impairment monitoringand response method that may be implemented by the vehicle occupantmonitoring system;

FIG. 7 illustrates a flow diagram of an exemplary inebriation monitoringand response method that may be implemented by the vehicle occupantmonitoring system;

FIG. 8 illustrates a flow diagram of an exemplary ergonomic monitoringand response method that may be implemented by the vehicle occupantmonitoring system;

FIG. 9 illustrates a flow diagram of an exemplary accident monitoringand response method that may be implemented by the vehicle occupantmonitoring system;

FIG. 10 illustrates a flow diagram of an exemplary unauthorized occupantmonitoring and response method that may be implemented by the vehicleoccupant monitoring system;

FIG. 11 illustrates a flow diagram of an exemplary security threatmonitoring and response method that may be implemented by the vehicleoccupant monitoring system;

FIG. 12 illustrates a flow diagram of an exemplary operator performancemonitoring and response method that may be implemented by the vehicleoccupant monitoring system;

FIG. 13 illustrates a flow diagram of an exemplary vehicle occupant riskdetermination method that may be implemented by the vehicle occupantmonitoring system; and

FIG. 14 illustrates a flow diagram of an exemplary mobile devicedisablement method that may be implemented by the vehicle occupantmonitoring system.

DETAILED DESCRIPTION

The systems and methods described herein may be used to monitor andrespond to abnormal or emergency situations that may occur within avehicle or during operation of a vehicle. Although such situations maybe described as “abnormal” or “emergency,” no implication of unusual oruncommon occurrences is intended. Instead, such terms are used todistinguish situations requiring a corrective response from situationsoccurring in the ordinary operation of the vehicle under ordinaryconditions following proper usage procedures. To detect and respond toabnormal or emergency situations, the systems and methods describedherein may collect sensor data regarding vehicle occupants, determinecharacteristics of the vehicle occupants (e.g., heart rate, pulsestrength, posture, temperature, etc.), compare the characteristicsagainst expected values, and determine whether the determinedcharacteristics indicate an abnormal situation. If an abnormal situationis determined to exist based upon the characteristics, the systems andmethods may determine and implement an appropriate response (e.g.,taking control of the vehicle, providing a notification to the vehicleoperator, initiating emergency communications, etc.).

Exemplary Vehicle Occupant Monitoring System

FIG. 1 illustrates a block diagram of an exemplary vehicle occupantmonitoring system 100. The high-level architecture includes bothhardware and software applications, as well as various datacommunications channels for communicating data between the varioushardware and software components. The vehicle occupant monitoring system100 may be roughly divided into front-end components 102 and back-endcomponents 104. The front-end components 102 monitor vehicle occupants,including a vehicle operator 106, for indications of abnormal situationsusing data from a variety of sensors within a vehicle 108 (e.g., a car,truck, etc.). The sensors may include one or more infrared (IR) sensors120, cameras 124, microphones 126, pressure sensors (not shown), orother similar sensor devices disposed within the vehicle 108(collectively, the “sensors”). In some embodiments, part or all of thesensors may be disposed within a mobile computing device 110, such as asmartphone. The sensors may be removably or permanently installed withinthe vehicle 108 and may communicate with the mobile device 110 or anon-board computer 114.

The front-end components 102 may further process the sensor datacollected from the one or more sensors using the mobile device 110 oron-board computer 114. When an abnormal or emergency situation isdetermined to exist, one or more appropriate responses to the situationmay be determined using the mobile device 110 or on-board computer 114.Such responses may include alerting the vehicle operator 106, presentingmitigating stimuli (e.g., music, massages, etc.), controlling vehicleoperation, initiating communication (e.g., telephone or texttransmissions to emergency services), or taking other actions to addressthe abnormal or emergency situation.

In some embodiments of the system, the front-end components 102 maycommunicate with the back-end components 104 via a network 130. Theback-end components 104 may use one or more servers 140 to process thesensor data provided by the front-end components 102, to store userprofiles created based upon sensor data, to determine and/or implementresponses to abnormal or emergency situations, or to perform otherfunctions of the system, as described herein. In some embodiments, thefront-end components 102 may form a stand-alone system that does notinclude the back-end components 104. Alternatively, in otherembodiments, the front-end components 102 may be implemented as athin-client system, with substantially all processing and data storageperformed by the server 140 using sensor data transmitted through thenetwork 130.

The front-end components 102 may be disposed within, or communicativelyconnected to, one or more mobile devices 110 or on-board computers 114,which may be permanently or removably installed in the vehicle 108. Themobile device 110 or the on-board computer 114 may interface with one ormore sensors within the vehicle 108, which sensors may also beincorporated within or connected to the mobile device 110 or theon-board computer 114. The one or more IR sensors 120 may includethermal imaging devices, IR scene projectors, or other IR sensor devicescapable of generating IR data. The one or more cameras 124 may includedigital cameras or other similar devices, such as charge-coupleddevices, to detect electromagnetic radiation in the visual range orother wavelengths. In some embodiments, the IR sensors 120 or cameras124 may include illumination devices to stimulate emission within atargeted range. The IR sensors 120 or cameras 124 may be disposed atvarious locations within the vehicle 108 to obtain a more complete viewof the vehicle cabin or passenger compartment. In a preferredembodiment, IR sensors 120 may be disposed at between six and tenseparate locations within the vehicle 108 to obtain optima coveragewithout unnecessary overlap between the sensor data from each location.In some embodiments, the system 100 may include various physiologicalsensors (not shown) in addition to the IR sensors 120 and cameras 124.Any of the sensors within the vehicle 108 may be installed by themanufacturer of the vehicle 108 or as an aftermarket modification to thevehicle 108. The mobile device 110 or the on-board computer 114 mayfurther interface with various output devices in the vehicle 108, suchas one or more speakers 122 or displays (not shown). The sensors mayalso include other sensors currently existing or later developed.

In some embodiments, the on-board computer 114 may supplement thefunctions performed by the mobile device 110 described herein. Inanother embodiment, the on-board computer 114 may perform all of thefunctions of the mobile device 110 described herein, in which case nomobile device 110 may be present in the system 100. In yet anotherembodiment, the mobile device 110 may perform all of the functions ofthe on-board computer 114, in which case no on-board computer 114 may bepresent in the system 100. The mobile device 110 or on-board computer114 may communicate with the network 130 over links 112 and 118,respectively. Additionally, the mobile device 110 and on-board computer114 may communicate with one another directly over link 116.

The on-board computer 114 may be a general-use on-board computer capableof performing many functions relating to vehicle operation or adedicated computer for monitoring vehicle occupants. Further, theon-board computer 114 may be installed by the manufacturer of thevehicle 108 or as an aftermarket modification to the vehicle 108. Themobile device 110 may, be either a general-use mobile personal computer,cellular phone, smart phone, tablet computer, or wearable device (e.g.,a watch, glasses, etc.) or a dedicated vehicle occupant monitoringdevice. In some embodiments, the mobile device 110 or on-board computer114 may be thin-client devices that outsource some or most of theprocessing to the server 140.

One or more vehicle operators 106 may operate the vehicle 108. Whileshown in a slightly reclined sitting position, those of ordinary skillin the art will appreciate that the vehicle operator 106 could besituated in any number of ways (e.g., reclining at a different angle,standing, etc.) and may operate the vehicle 108 using controls otherthan the steering wheel and pedals shown in FIG. 1 (e.g., one or moresticks, yokes, levers, etc.). Additionally, one or more additionaloccupants (not shown) may be passengers within the vehicle 108. Theysystem 100 may monitor the characteristics and/or activity of vehicleoperators 106 and other occupants of the vehicle 108.

One or more feedback devices 128 may be included within the vehicle 108.Such feedback devices 128 may include massage devices, heaters, coolers,or other similar devices, which may be disposed within a seat, steeringwheel, or other portions of the vehicle 108. The one or more feedbackdevices 128 may be communicatively connected to and controlled by themobile device 110 or the on-board computer 114.

In some embodiments, the front-end components 102 may communicate withthe back-end components 104 via the network 130. The network 130 may bea proprietary network, a secure public internet, a virtual privatenetwork or some other type of network, such as dedicated access lines,plain ordinary telephone lines, satellite links, cellular data networks,combinations of these, etc. Where the network 130 comprises theInternet, data communications may take place over the network 130 via anInternet communication protocol.

The back-end components 104 include one or more servers 140. Each server140 may include one or more computer processors 162 adapted andconfigured to execute various software applications and components ofthe vehicle occupant monitoring system 100, in addition to othersoftware applications. The server 140 may further include a database146, which may be adapted to store data related to the operation of thevehicle occupant monitoring system 100. Such data might include, forexample, user profiles, images, sensor outputs, data analyzed accordingto the methods discussed below, or other kinds of data pertaining to thevehicle occupants that has been uploaded to the server 140 via thenetwork 103. The server 140 may access data stored in the database 146when executing various functions and tasks associated with the operationof the vehicle occupant monitoring system 100.

The server 140 may have a controller 155 that is operatively connectedto the database 146 via a link 156. It should be noted that, while notshown, additional databases may be linked to the controller 155 in aknown manner. The controller 155 may include a program memory 160, aprocessor 162 (which may be called a microcontroller or amicroprocessor), a random-access memory (RAM) 164, and an input/output(I/O) circuit 166, all of which may be interconnected via anaddress/data bus 165. It should be appreciated that although only onemicroprocessor 162 is shown, the controller 155 may include multiplemicroprocessors 162. Similarly, the memory of the controller 155 mayinclude multiple RAMS 164 and multiple program memories 160. Althoughthe 0 circuit 166 is shown as a single block, it should be appreciatedthat the I/O circuit 166 may include a number of different types of I/Ocircuits. The RAM 164 and program memories 160 may be implemented assemiconductor memories, magnetically readable memories, or opticallyreadable memories, for example. The controller 155 may also beoperatively connected to the network 130 via a link 135.

The server 140 may further include a number of software applicationsstored in a program memory 160. The various software applications mayinclude a monitoring application 142 for processing sensor data,determining occupant characteristics, determining abnormal situations,and/or generating user profiles using the processor 162 of the server140. The software applications may further include a responseapplication 143 for determining and/or causing implementation ofresponses to abnormal situations.

Although the vehicle occupant monitoring system 100 is shown to includeone mobile device 110, one on-board computer 114, and one server 140, itshould be understood that different numbers of mobile devices 110,on-board computers 114, and servers 140 may be utilized. For example,the system 100 may include a plurality of servers 140 and hundreds ofmobile devices 110 or on-board computers 114, all of which may beinterconnected via the network 130. Furthermore, the database storage orprocessing performed by the one or more servers 140 may be distributedamong a plurality of servers 140 in an arrangement known as “cloudcomputing.” This configuration may provide various advantages, such asenabling near real-time uploads and downloads of information as well asperiodic uploads and downloads of information. This may in turn supporta thin-client embodiment of the mobile device 110 or on-board computer114.

Exemplary Computing Device

FIG. 2 illustrates a block diagram of an exemplary mobile device 110 oran on-board computer 114 for use in the vehicle occupant monitoringsystem 100. Part or all of the sensor data may come from sensorsincorporated within or connected to the mobile device 110 or on-boardcomputer 114. Additionally, or alternatively, the communication unit 220may receive sensor data from one or more external sensors within thevehicle 108. The sensor data may be processed by the controller 204 todetermine information regarding the occupants within the vehicle 108.When the controller 204 determines that an abnormal situation exists,appropriate responses may be determined using the controller 204 basedupon the type of abnormal situation identified by the sensor data.Different types of abnormal situations and appropriate responses aredescribed in further detail below. The mobile device 110 or on-boardcomputer 114 may then control the implementation of the response. Insome instances, this may include presenting alerts to the vehicleoperator 108 using the display 202, speakers 122 or 246, feedbackdevices 128, and/or other appropriate output devices (not shown).Additionally, or alternatively, the mobile device 110 or on-boardcomputer 114 may transmit the sensor data to the server 140 forprocessing or may receive responses determined by the server 140 forimplementation within the vehicle 108 via the network 130.

The mobile device 110 or on-board computer 114 may include a display202, a Global Positioning System (GPS) unit 206, a communication unit220, a front image capture device 218, a back image capture device 222,an accelerometer array 224, one or more additional sensors (not shown),a user-input device (not shown), a speaker 246, and, like the server140, a controller 204. In some embodiments, the mobile device 110 andon-board computer 114 may be integrated into a single device, or eithermay perform the functions of both. Functions performed by either themobile device 110 or the on-board computer 114 may also be performed bythe mobile device 110 in concert with the on-board computer 114.

Similar to the controller 155, the controller 204 includes a programmemory 208, one or more microcontrollers or microprocessors (MP) 210, aRAM 212, and an I/O circuit 216, all of which are interconnected via anaddress/data bus 214. The program memory 208 includes an operatingsystem 226, a data storage 228, a plurality of software applications230, and a plurality of software routines 234. The operating system 226,for example, may include one of a plurality of mobile platforms such asthe iOS®, Android™, Palm® webOS, Windows® Mobile/Phone, BlackBerry® OS,or Symbian® OS mobile technology platforms, developed by Apple Inc.,Google Inc., Palm Inc. (now Hewlett-Packard Company), MicrosoftCorporation, Research in Motion (RIM), and Nokia, respectively. The datastorage 228 may include data such as user profiles and preferences,application data for the plurality of applications 230, routine data forthe plurality of routines 234, and other data necessary to interact withthe server 140 through the network 130. In some embodiments, thecontroller 204 may also include, or otherwise be communicativelyconnected to, other data storage mechanisms (e.g., one or more hard diskdrives, optical storage drives, solid state storage devices, etc.) thatreside within the mobile device 110 or on-board computer 114.

As discussed with reference to the controller 155, it should beappreciated that although FIG. 2 depicts only one microprocessor 210,the controller 204 may include multiple microprocessors 210. Similarly,the memory of the controller 204 may include multiple RAMs 212 andmultiple program memories 208. Although the FIG. 2 depicts the I/Ocircuit 216 as a single block, the I/O circuit 216 may include a numberof different types of I/O circuits. The controller 204 may implement theRAMs 212 and the program memories 208 as semiconductor memories,magnetically readable memories, or optically readable memories, forexample.

The communication unit 220 may communicate with one or more externalsensors within the vehicle 108 (including IR sensors 120 and/or cameras124), mobile devices 110, on-board computers 114, or servers 140 via anysuitable wireless communication protocol network, such as a wirelesstelephony network (e.g., GSM, CDMA, LTE, etc.), a Wi-Fi network (802.11standards), a WiMAX network, a Bluetooth network, etc. Additionally, oralternatively, the communication unit 220 may also be capable ofcommunicating using a near field communication standard (e.g., ISO/IEC18092, standards provided by the NFC Forum, etc.). Furthermore, thecommunication unit 220 may provide input signals to the controller 204via the I/O circuit 216. The communication unit 220 may also transmitsensor data, device status information, control signals, or other outputfrom the controller 204 to one or more external sensors within thevehicle 108, mobile devices 110, on-board computers 114, or servers 140.In some embodiments, the communication unit 220 of the on-board computer114 may communicate (via a wired connect, Bluetooth, NFC, etc.) with thecommunication unit 220 of the mobile device 110 to establish acommunications link between the two devices. This may be particularlyadvantageous in embodiments in which the mobile device 110 is asmartphone associated with the vehicle operator 106 or other vehicleoccupants, which smartphone may utilize the speakers, microphones,and/or displays installed within the vehicle 108.

The GPS unit 206 may use “Assisted GPS” (A-GPS), satellite GPS, or anyother suitable global positioning protocol (e.g., the GLONASS systemoperated by the Russian government) or system that locates the positionof the mobile device 110 or on-board computer 114. For example, A-GPSutilizes terrestrial cell phone towers or Wi-Fi hotspots (e.g., wirelessrouter points) to more accurately and more quickly determine location ofthe mobile device 110 or on-board computer 114, while satellite GPSgenerally is more useful in remote regions that lack cell towers orWi-Fi hotspots.

The one or more IR sensors 120 and/or cameras 124 may include the imagecapture devices 218 or 222. The front and back image capture devices 218and 222 may be built-in cameras within the mobile device 110 or on-boardcomputer 114. Additionally, or alternatively, they may be peripheralcameras, such as webcams, dashcams, or other cameras installed inside oroutside the vehicle 108 that are communicatively coupled with the mobiledevice 110 or on-board computer 114. The front image capture device 218may be oriented toward the vehicle operator 106 to observe the vehicleoperator 106 as described below. The back image capture device 222 maybe oriented toward the front of the vehicle 108 to observe a road, lanemarkings, or other objects in front of the vehicle 108. Some embodimentsmay have both a front image capture device 218 and a back image capturedevice 222, but other embodiments may have only one or the other.Further, either or both of the front image capture device 218 and backimage capture device 222 may include an infrared illuminator 218 i, 222i, respectively, or other device to facilitate low light or night imagecapturing. Such infrared illuminators 218 i and 222 i may beautomatically activated when light is insufficient for image capturing.

The accelerometer array 224 may include one or more accelerometerspositioned to determine the force and direction of movements of themobile device 110 or on-board computer 114. In some embodiments, theaccelerometer array 224 may include an X-axis accelerometer 224 x, aY-axis accelerometer 224 y, and a Z-axis accelerometer 224 z to measurethe force and direction of movement in each dimension respectively. Itwill be appreciated by those of ordinary skill in the art that a threedimensional vector describing a movement of the mobile device 110 oron-board computer 114 through three dimensional space can be establishedby combining the outputs of the X-axis, Y-axis, and Z-axisaccelerometers 224 x, y, z using known methods.

Furthermore, the mobile device 110 or on-board computer 114 may alsoinclude (or be coupled to) other sensors such as a thermometer,microphone, thermal image capture device, electroencephalograph (EEG),galvanic skin response (GSR) sensor, heart rate sensor, other biometricsensors, etc. Physiological sensor data may be used to measureindications that the vehicle operator 106 is impaired, experiencing amedical emergency, or experiencing another abnormal condition. Athermometer or thermal image capture device may be used to determine anabnormal body temperature or a change in body temperature of the vehicleoperator 106 that may indicate stress or drowsiness, for example. Amicrophone may be used to receive voice inputs, and may also be used todetect irregularities in the voice of the vehicle operator 106indicating that vehicle operator 106 is agitated or under stress. An EEGmay be used to determine whether a vehicle operator 106 is stressed,distracted, or otherwise impaired. A GSR sensor may be used to detectwhether the vehicle operator 106 is stressed (i.e., that the conductanceof the vehicle operator's 106 skin has varied from its normal level).Other biometric sensors may similarly be used to detect whether avehicle operator 106 is in an impaired state.

The mobile device 110 or on-board computer 114 may include a user-inputdevice (not shown). The user-input device may include a “soft” keyboardthat is displayed on the display 202 of the mobile device 110 oron-board computer 114, an external hardware keyboard communicating via awired or a wireless connection (e.g., a Bluetooth keyboard), an externalmouse, a microphone, or any other suitable user-input device. Theuser-input device may also include a microphone capable of receivinguser voice input, such as the microphone 126.

The one or more processors 210 may be adapted and configured to executeany of one or more of the plurality of software applications 230 or anyone or more of the plurality of software routines 234 residing in theprogram memory 204, in addition to other software applications. One ofthe plurality of applications 230 may be a monitoring application 232that may be implemented as a series of machine-readable instructions forperforming the various tasks associated with implementing part or all ofthe data collection and assessment functions of the vehicle occupantmonitoring system 100. One of the plurality of applications 230 may be aresponse application 236 that may be implemented as a series ofmachine-readable instructions for determining and implementing anappropriate response to an abnormal situation. Another application ofthe plurality of applications may include a communication application242 that may be implemented as a series of machine-readable instructionsfor sending and receiving electronic communications through the network130, including telephonic communications. One of the plurality ofroutines may include an image capture routine 238 that coordinates withthe front image capture device 218 or back image capture device 222 toretrieve image data for use with one or more of the plurality ofapplications, such as the monitoring application 232, or for use withother routines. Another routine in the plurality of routines may includea profile access routine 240 that retrieves, modifies, and/or storesuser profiles in the data storage 228 of the mobile device 110 oron-board computer 114 or the database 146 of the server 140.

A user may launch the monitoring application 232 from the mobile device110 or on-board computer 114 in order to initiate operation of thevehicle occupant monitoring system 100 to monitor and respond toabnormal or emergency situations. Additionally, or alternatively, thevehicle occupant monitoring system 100 may automatically beginmonitoring the vehicle occupants when the vehicle 108 is started. Insome embodiments, the vehicle occupant monitoring system 100 mayautomatically monitor the vehicle 108 for occupants on a continuousbasis, regardless of whether the vehicle 108 is in use or is not in use.

In embodiments where the mobile device 110 or on-board computer 114 is athin-client device, the server 140 may perform many of the processingfunctions remotely that would otherwise be performed by the mobiledevice 110 or on-board computer 114. In such embodiments, the mobiledevice 110 or on-board computer 114 may gather data from its sensors orother sensors as described herein. Rather than analyzing the datalocally, however, the mobile device 110 or on-board computer 114 mayinstead send the data to the server 140 for remote processing. Theserver 140 may perform the analysis of the gathered data to determinewhether an abnormal situation exists. If the server 140 determines thatan abnormal situation exists, the server 140 may determine one or moreappropriate responses to the situation. The server 140 may then commandthe mobile device 110 or on-board computer 114 to implement the one ormore responses, as described below. Additionally, the server 140 maygenerate metrics and suggestions regarding vehicle usage based on thegathered data.

Exemplary Data Collection

Each authorized vehicle operator 106 or usual passenger may have anassociated user profile created by the vehicle occupant monitoringsystem 100. The user profile may include information relating to theusual characteristics of the user, such as facial features, restingheart rate, or vocal patterns. The user profile may further includeinformation regarding the user's connection with the vehicle (e.g., aninsured driver, an authorized passenger, an unknown occupant, etc.). Insome embodiments, the profile may further include information regardinga mobile device 110 associated with the user, such as a smartphone. Theuser profile may be used by the vehicle occupant monitoring system 100during monitoring to distinguish between occupants within the vehicle108 and/or to determine abnormal situations based upon deviations fromuser characteristics or preferences stored in the user profile.

FIG. 3 illustrates a flow diagram of an exemplary embodiment of a userprofile generation method 300 for creating or updating a user profile.The method 300 may be implemented by the mobile device 110 or on-boardcomputer 114 using the sensors disposed within the vehicle 108 to obtainand process sensor data associated with a user. The method 300 may beimplemented for each of a plurality of occupants of the vehicle 108 tocreate a new user profile or update a user profile for each occupant.The user profiles may then be stored in the data storage 228 of themobile device 110 or on-board computer 114 or may be transmitted throughthe network 130 to be stored in the database 146. Although the method300 is described below as being implemented using the on-board computer114, some or all of the steps may likewise be implemented using themobile device 110, the server 140, or a combination of some or all ofthese.

At block 302, the on-board computer 114 may receive a command to createor to update a user profile associated with a user who is an occupant ofthe vehicle 108. The command may be entered by the user, in someembodiments, or the command may be automatically generated by theon-board computer 114. For example, the on-board computer 114 mayautomatically implement the method 300 upon determining the presence ofan occupant in the vehicle 108 for whom no associated user profile canbe located in the data storage 228 or database 146. The method 300 maycontinue to be implemented during vehicle operation or across multiplevehicle trips until sufficient data has been collected to ensurereliability and accuracy of the user profile.

At block 304, the on-board computer 114 may receive sensor data fromsome or all of the sensors within the vehicle 108. Specifically, thereceived sensor data may include data from one or more IR sensors 120,cameras 145, and/or microphones 126. In some embodiments, the sensors orthe on-board computer 114 may process the raw sensor data to produce ageneralized data set that is more directly usable in the followingblocks 306-314. The IR sensor data may include a point cloud of datapoints in three dimensions, derived from two-dimensional images receivedat the one or more IR sensors 120 and data values indicating distancefrom the IR sensors 120. For example, IR scene projection techniques maybe implemented by the IR sensors 120 or the on-board computer 114 toproduce a projection of the scene observed by the one or more IRsensors. Similarly, image from a plurality of cameras 124 may becombined to produce a three-dimensional image, such as may be used infacial recognition techniques.

At block 306, the on-board computer 114 may determine user facialcharacteristics from the sensor data. This may involve combining datafrom a plurality of sensors of the same or different types (e.g.,combining data from IR sensors 120 and cameras 124). Facial recognitionalgorithms and processes may be used to determine salient facialfeatures uniquely (or with a high degree of statistical certainty)associated with the user. Such features may be related to the shape,size, angle, color, contours, or relative dimensions of the eyes, nose,forehead, mouth, chin, ears, or face as a whole.

At block 308, the on-board computer 114 may determine user vocalcharacteristics using sensor data from the microphone 126. Such vocalcharacteristics may include the pitch, volume, timbre, duration, orpatterns of the voice or speech of the user. Frequency analysis may, beperformed by the on-board computer 114 to determine a voice print orsub-profile indicating information regarding the user's vocalcharacteristics, which may be used to identify the user.

At block 310, the on-board computer 114 may determine user heart ratecharacteristics and/or breathing characteristics. Sensor data from theIR sensors 120 and/or cameras 124 may be used to determine minor changesin the flushness of the user's skin to identify systolic and diastolicpoints within the cardiac cycle of the user. Such cyclical changes maybe identified by changes in volume, temperature, or color in the sensordata. In some embodiments, the microphone 126 may also be used toidentify hear rate by sound. In addition to the heart rate, the patternof expansion and contraction associated with the cardiac cycle of theuser may be determined as a user characteristic associated with the userheartbeat. User breathing characteristics may similarly be determinedbased upon sensor data received from the sensors within the vehicle 108.The user breaching characteristics may include respiration rate,intensity, volume, pattern, sound pattern, and/or composition. Forexample, the IR sensor data may be used to determine the moisturecontent of the user's breath.

At block 312, the on-board computer 114 may determine user skeletalcharacteristics based upon the sensor data. This may include using theIR data to identify user joints and skeletal segments between thejoints. In some embodiments, this may include observing user movementover a period of time to identify joints and segments, which may beconnected in a computer model to form a functional approximate of theuser's skeleton. This may further include determining the relativeand/or absolute sizes and positions of the user's head, neck, torso, andlimbs. Additionally, normal ranges of motion may be determined forjoints. In further embodiments, information regarding the user's posturemay be determined as part of the user skeletal characteristics.

At block 314, the on-board computer 114 may further determine usermuscular characteristics. This may involve modeling the user'smusculature based in part upon the model of the user's skeletondeveloped at block 312. The sensor data may be used to determine musclegroups or conditions of the user, particularly temperature or flushness(which may indicate an increased level of blood flow to the muscles of aregion of the user's body). Such characteristic data may be useful inlater determining user fatigue or illness.

At block 316, the on-board computer 114 may determine a user profile (oran update to an existing user profile) based upon the determined usercharacteristics. In some embodiments, further user characteristics maybe determined, such as movement characteristics, that may also beincluded in the user profile. In further embodiments, additionalinformation pertaining to the user may be included in the user profile,along with the user characteristics. For example, an indication of amobile phone associated with the user may be included, which mayfacilitate management of the user's phone based upon whether the user isoperating the vehicle or riding in the vehicle as a passenger. Asanother example, medical information or emergency contact informationassociated with the user may be included in the user profile, in case ofmedical or other emergency. In some embodiments, information regardinguser authorization with respect to one or more vehicles may be includedin the user profile.

Exemplary Abnormal Situation Detection

The vehicle occupant monitoring system 100 may be used to monitor theoccupants of the vehicle 108 during vehicle operation to determinewhether an abnormal situation has occurred. In some embodiments,monitoring may continue even when the vehicle is not in operation. If anabnormal situation is identified, the vehicle occupant monitoring system100 may then take appropriate actions to respond to the situation. Insome embodiments, user profiles may be used to determine abnormalsituations by comparing sensor data obtained during monitoring withnormal ranges for occupants of the vehicle 108 established in the userprofiles.

FIG. 4 illustrates a flow diagram of an exemplary embodiment of amonitoring and response method 400 that may be implemented by thevehicle occupant monitoring system 100. At block 402, the method maybegin with receiving a command to begin monitoring the occupants of thevehicle 108. Sensor data may be collected at block 404 and compared withprofiles accessed at block 406 to identify one or more vehicle occupantsat block 408. Unknown vehicle occupants for whom matching user profilescould not be found may be identified as such at block 410, and newprofiles may be created for such unknown occupants in some embodiments.At blocks 412-424, the vehicle occupants may be monitored using sensordata to determine whether any abnormal situations arise. Sensor data maybe collected at block 412, from which current occupant characteristicsmay be determined at block 414. The current occupant characteristics maybe compared against user profiles or other known patterns at block 416to determine whether an abnormal situation exists. When an abnormalsituation is determined to exist at block 418, an appropriate responsemay be determined and implemented, respectively, at blocks 420 and 422.At block 424, the method 400 may determine whether to continuemonitoring the vehicle occupants. If monitoring will continue, themethod 400 may continue at block 412 by collecting further sensor data.If monitoring is finished, the method 400 may terminate. Although themethod 400 is described below as being implemented using the on-boardcomputer 114, some or all of the steps may likewise be implemented usingthe mobile device 110, the server 140, or a combination of some or allof these.

At block 402, the on-board computer 114 may receive a command tomonitoring occupants within the vehicle 108. The command may be enteredby the user, in some embodiments, or the command may be automaticallygenerated by the on-board computer 114. For example, the on-boardcomputer 114 may automatically implement the method 400 upon determiningthe presence of an occupant in the vehicle 108 or when the vehicle isstarted. The method 400 may continue to be implemented while the vehicleremains in operation. In some embodiments, the method 400 may continueto monitor any occupants within the vehicle even after vehicle operationhas concluded.

At block 404, the on-board computer 114 may collect sensor data from thesensors within the vehicle. In particular, sensor data may be collectedfrom one or more IR sensors 120, cameras 124, and/or microphones 126.The sensor data may be collected for a short period of time, which maybe taken as a snapshot of the vehicle occupants. Based upon the sensordata, the on-board computer 114 may determine a number of occupants ofthe vehicle, including types of occupants e.g., child, infant, adult,etc.). In some embodiments, the on-board computer 114 may process partor all of the received sensor data to determine occupant characteristicsfor comparison against occupant characteristics stored in user profiles,as discussed above.

At block 406, the on-board computer 114 may access one or more userprofiles stored in the data storage 228 or the database 146. The userprofiles may be selected from a set of user profiles associated with thevehicle 108. Additionally, or alternatively, the user profiles may besearched based upon the sensor data collected at block 404 to findmatches.

At block 408, the on-board computer 114 may identify one or more vehicleoccupants as users based upon the user profiles. This may includeregressing the sensor data or derived occupant characteristics againstdata stored in a plurality of user profiles to determine a probabilityof a match between one or more occupants and one or more user profiles.If no match can be found in the user profiles accessed at block 406, theon-board computer 114 may, attempt to find a match with additional userprofiles stored in the system memory 228 or the database 146.Additionally, or alternatively, the on-board computer 114 may collectfurther sensor data and attempt to determine the identities of the oneor more vehicle occupants using the new sensor data. If the identity ofone or more of the vehicle occupants cannot be determined withsufficient certainty, then the on-board computer 114 may identify suchoccupants as unknown occupants at block 410.

At block 410, the on-board computer 114 may identify as unknownoccupants any vehicle occupants who could not be identified at block408. In some embodiments, the presence of unknown occupants may beconsidered an indicator of an abnormal situation. Iii furtherembodiments, the on-board computer 114 may generate a new user profilefor each unknown occupant according the method 300 above. Sensor data(including images) may be automatically uploaded to the server 140 andstored in the database 146 for any unknown occupants, in someembodiments. This may facilitate loss recovering the case of theft andmay serve as a deterrent.

At blocks 412-424, the on-board computer 114 may continue monitoring thevehicle occupants (and responding to any abnormal situations that aredetected) until the method 400 terminates. At block 412, the on-boardcomputer 114 may collect further sensor data from the one or moresensors disposed within the vehicle 108. Sensor data from one or more IRsensors 120, cameras 124, microphones 126, or other sensors within thevehicle 108 may be continuously or periodically obtained, processed,and/or stored by the on-board computer 114. In some embodiments,processed or raw sensor data may be transmitted to the server 140 viathe network 130. The server 140 may then process and/or store the dataor information derived therefrom in the database 146. In someembodiments, part or all of the data or information derived therefrommay be further communicated from the server 140 to one or more thirdparties (such as emergency services) via the network 130.

At block 414, the on-board computer 114 may process the data todetermine occupant characteristics associated with one or more usersand/or other occupants of the vehicle 108. As discussed above, thedetermined occupant characteristics may include facial characteristics,vocal characteristics, heart rate characteristics, breathingcharacteristics, skeletal characteristics, muscular characteristics,and/or other characteristics associated with each occupant.

At block 416, the on-board computer 114 may determine whether anabnormal situation exists based upon the occupant characteristicsdetermined at block 414. The on-board computer 114 may compare thedetermined occupant characteristics with information retrieved from theuser profile associated with the occupant to determine whether anabnormal situation exists. This may include determining whether one ormore occupant characteristics determined from the sensor data areoutside a range of normal values for such occupant characteristics basedupon the user profile associated with the occupant. In some embodiments,information regarding a plurality of occupant characteristics or aplurality of occupants may be used to determine whether an abnormalsituation exists. Based upon the sensor data, occupant characteristics,and/or user profiles, more than one type of abnormal situation may beidentified for each of one or more vehicle occupants in some instances.Examples of determining and responding to various types of abnormalsituations are presented below with FIGS. 5-13.

At block 418, the on-board computer 114 may determine whether one ormore abnormal situations have been determined to exist at block 416. Ifno abnormal situations are determined to exist, the method 400 maycontinue at block 424. If at least one abnormal situation has beendetermined to exist, the on-board computer 114 may determine one or moreappropriate responses to the abnormal situation at block 420.

In determining an appropriate response at block 420, the on-boardcomputer 114 may base the response, at least in part, upon whether theabnormal situation relates to the vehicle operator 106 or a passengerwithin the vehicle 108, as well as whether there are passengers withinthe vehicle. For example, the on-board computer 114 may determine topresent an alert regarding an abnormal situation involving a passenger,whereas the same abnormal situation would result in a determination toadjust or control operation of the vehicle 108 if it were to involve thevehicle operator 106. Similarly, the appropriate response determinationmay be dependent upon whether unknown occupants are in the vehicle orwhether the abnormal situation involves one or more unknown occupants.User preferences stored in the user profiles may also be used indetermining appropriate responses to abnormal situations.

The on-board computer 114 may further determine risk levels associatedwith a plurality of possible responses, which risk levels may becompared to determine one or more responses that reduce or minimizerisk. In some embodiments, the on-board computer 114 may, determine oneor more responses to reduce risk levels below an acceptable limit orthreshold, which may include determining responses having the lowestcost or that are the least intrusive that are determined to accomplishsuch risk reduction to acceptable levels. In further embodiments, thedetermined response to an abnormal situation may include taking noaction or simply continuing to monitor the vehicle occupants.

Once one or more appropriate responses to one or more abnormalsituations are determined at block 420, the on-board computer 114 mayimplement the determined response or cause the determined responses tobe implemented at block 422. This may include adjusting or controllingoperation of the vehicle 108, such as turning on an autonomous oradaptive cruise control functionality of the vehicle 108 or piloting andparking the vehicle 108 in a safe location. Similarly, the on-boardcomputer 114 may adjust the environment of the vehicle 108 to influenceone or more occupants, such as by adjusting the temperature, openingwindows, selecting music, or adjusting lighting levels within thevehicle 108. This may also include presenting alerts or warnings to oneor more occupants, using speakers 122 or 246, displays 202, feedbackdevices 128, and/or mobile devices 110.

In some instances, the on-board computer 114 may cause telephonic orelectronic communications to be initiated, such as by contacting anemergency service switchboard. In further embodiments, the appropriateresponse may include determining an adjustment to an insurance policy,such as an adjustment to a rating, premium, deductible, discount, orsurcharge. Upon implementing the determined response or responses, themethod 400 may continue at block 424.

In some embodiments, the on-board computer 114 may determine to provideaccess to sensor data or determined occupant characteristic data to athird party at block 420. For example, access to heart ratecharacteristics may be provided to medical professionals or an emergencyresponse service if an occupant is determined to be experiencing amedical emergency. Such access may be determined based upon userpreferences in a user profile or other information associated with thevehicle occupant. In some embodiments, this may include allowingemergency responders to locate and request access to relevant sensor oroccupant characteristic information based upon a telephone number,location of the vehicle, or other identifying information. To implementsuch response at block 422, the on-board computer 114 may utilize thecommunication unit 220 of the on-board computer 114 or mobile device 110to communicate through the network 140.

At block 424, the on-board computer 114 may determine whether monitoringshould continue. In some embodiments, this may include determiningwhether the vehicle 108 is in user or whether any occupants are detectedwithin the vehicle 108. In further embodiments, the method 400 maycontinue until manually terminated by an authorized user. In someembodiments, the occurrence of one or more events may cause the method400 to terminate and restart at block 402. Such events may includeengine ignition or vehicle start-up, entrance or exit of one or morevehicle occupants, or exceeding a time threshold of continuous operationof the method 400. If the on-board computer 114 determines thatmonitoring should continue at block 424, the method 400 may continue atblock 412. If the on-board computer 114 determines that monitoringshould not continue at block 424, the method 400 may terminate.

As noted above, the exemplary vehicle occupant monitoring system 100 andmonitoring and response method 400 may determine and respond to avariety of abnormal situations. The following examples indicate some ofthese various abnormal situations and appropriate responses that may bedetected and addressed by the system 100 and the method 400. Each of thefollowing FIGS. 5-13 illustrates an exemplary embodiment of a monitoringand response method that may replace blocks 412-422 in the method 400described above. Alternatively, each of the methods of FIGS. 5-13 may beindependently implemented within a vehicle, such as by implementationusing the exemplary vehicle occupant monitoring system 100. Although themethods 500-1200 are described below as being implemented using theon-board computer 114, some or all of the steps may likewise beimplemented using the mobile device 110, the server 140, or acombination of some or all of these.

Exemplary Medical Emergency Monitoring & Response

FIG. 5 illustrates a flow diagram of an exemplary medical emergencymonitoring and response method 500 that may be implemented by thevehicle occupant monitoring system 100. The method 500 may monitorvehicle occupants to determine when a medical emergency is occurring andrespond appropriately.

At block 512, the on-board computer 114 may collect sensor data from oneor more sensors within the vehicle 108. Data from IR sensors 120 and/orcameras 124 may be particularly relevant for determining usercharacteristics associated with medical health or medical emergencies.

At block 514, the on-board computer 114 may determine one or more useror occupant characteristics associated with medical health or medicalemergencies. User characteristics such as heart rate, pulse strength,heartbeat pattern, breathing rate, breathing volume, breathing pattern,facial features, vocal pattern, and/or skin temperature may beparticularly relevant, though other characteristics may likewise bedetermined.

At block 516, the on-board computer 114 may compare the determinedcharacteristics against user profile data to determine whether a medicalemergency exists. For example, a heart rate that suddenly rises above orfalls below a normal range for a vehicle occupant may be indicative of aheart attack or other acute medical condition, particularly if breathingor skin temperature characteristics likewise deviate from normal ranges.If no baseline user characteristics exist in a user profile for anoccupant (such as an unknown occupant), a generic user profile may beused as a baseline. In such instances changes in occupantcharacteristics may be used to determine the existence of a medicalemergency.

At block 518, the on-board computer 114 may determine whether anabnormal condition associated with a medical emergency has beendetermined to exist. If no such abnormal situation is found, the method500 may terminate. If such an abnormal situation is found, the on-boardcomputer may determine an appropriate response at block 520. Theappropriate response may be determined based upon the received sensordata, determined occupant characteristics, the presence or absence ofother occupants within the vehicle 108, and/or the nature and severityof the medical emergency. For example, an appropriate response to asevere medical emergency such as a heart attack of the vehicle operator106 may include causing the vehicle 108 to pull out of traffic andinitiate a telephone call with a medical emergency service.

In the same situation where a passenger within the vehicle 108 (ratherthan the vehicle operator 106) is having a heart attack, the on-boardcomputer 114 may present an alert to the vehicle operator 106, alongwith directions to an appropriate medical facility based upon thevehicle's location (e.g., a hospital or emergency care center). If aless severe or acute medical emergency exists, the on-board computer 114may provide an indication or alert to the occupants, such as bypresenting an alert using the speaker 122 or 246, display 202, orfeedback device 128, which may provide a haptic alert to the vehicleoperator 106 or other occupant.

Once one or more appropriate responses have been determined, theon-board computer 114 may implement the determined responses at block522 using the speaker 122 or 246, display 202, feedback device 128,communication unit 220, or other components of the system 100. Themethod 500 may then terminate.

Exemplary Impairment Monitoring & Response

FIG. 6 illustrates a flow diagram of an exemplary impairment monitoringand response method 600 that may be implemented by the vehicle occupantmonitoring system 100. The method 600 may monitor vehicle occupants todetermine when a vehicle operator 106 is impaired and respondappropriately. Such impairments may include physical impairments (e.g.,medical emergencies, drowsiness, inebriation, etc.) or emotionalimpairments (e.g., rage, distraction, anxiety, etc.). In someembodiments, impairments of passengers within the vehicle 108 maysimilarly be determined.

At block 612, the on-board computer 114 may collect sensor data from oneor more sensors within the vehicle 108. Data from IR sensors 120 and/orcameras 124 may be particularly relevant for determining usercharacteristics associated with operator impairments.

At block 614, the on-board computer 114 may determine one or more useror occupant characteristics associated with user impairments. Usercharacteristics such as heart rate, pulse strength, heartbeat pattern,breathing rate, breathing volume, breathing pattern, facial features,vocal pattern, skin temperature, posture, movement, and/or interactionwith items or occupants within the vehicle may be particularly relevant,though other characteristics may likewise be determined.

At block 616, the on-board computer 114 may compare the determinedcharacteristics against user profile data to determine whether the useris impaired. For example, a slowly declining heart rate or breathingrate relative to the user's normal range, coupled with head drooping orlack of eye movement, may indicate that the vehicle operator is becomingdrowsy. If no baseline user characteristics exist in a user profile foran occupant (such as an unknown occupant), a generic user profile may beused as a baseline. In such instances changes in occupantcharacteristics may be used to determine the existence of an impairment.

At block 618, the on-board computer 114 may determine whether anabnormal condition associated with a vehicle operator impairment hasbeen determined to exist. If no such abnormal situation is found, themethod 600 may terminate. If such an abnormal situation is found, theon-board computer may determine an appropriate response at block 620.The appropriate response may be determined based upon the receivedsensor data, determined occupant characteristics, the presence orabsence of other occupants within the vehicle 108, and/or the nature ofthe impairment. For example, an appropriate response to a drowsy vehicleoperator 106 may include playing upbeat music via the speaker 122 or246, increasing lighting conditions within the vehicle cabin, and/orproviding an alert to the vehicle operator 106 or other vehicleoccupants. Alternatively, an appropriate response for an anxious orenraged vehicle operator 106 may include playing soothing music,adjusting the cabin air temperature, and/or operating a feedback device128 to massage the vehicle operator 106.

If more severe or acute impairment exists, the on-board computer 114 mayassume control of vehicle operation and/or stop the vehicle at a safelocation out of traffic. In some embodiments, the on-board computer 114may prevent the vehicle 108 from starting to operate if the operator isdetermined to be impaired. Additionally, or alternatively, the on-boardcomputer 114 may provide an indication or alert to the occupants, suchas by presenting an alert using the speaker 122 or 246, display 202, orfeedback device 128, which may provide a haptic alert to the vehicleoperator 106 or other occupant.

Once one or more appropriate responses have been determined, theon-board computer 114 may implement the determined responses at block622 using the speaker 122 or 246, display 202, feedback device 128,communication unit 220, or other components of the system 100. Themethod 600 may then terminate.

exemplary INEBRIATION MONITORING & RESPONSE

FIG. 7 illustrates a flow diagram of an exemplary inebriation monitoringand response method 700 that may be implemented by the vehicle occupantmonitoring system 100. The method 700 may monitor vehicle occupants todetermine when a vehicle operator 106 is inebriated and respondappropriately. In some embodiments, inebriation of passengers within thevehicle 108 may similarly be determined.

At block 712, the on-board computer 114 may collect sensor data from oneor more sensors within the vehicle 108. Data from IR sensors 120 and/orcameras 124 may be particularly relevant for determining usercharacteristics indicative of inebriation.

At block 714, the on-board computer 114 may determine one or more useror occupant characteristics associated with inebriation. Usercharacteristics such as breath composition, vocal patterns, skintemperature, posture, movement, and/or interaction with items oroccupants within the vehicle may be particularly relevant, though othercharacteristics may likewise be determined. Of particular interest is IRrefraction caused by alcohol molecules exhaled by the vehicle operator106, which may be measured using the one or more IR sensors 120.

At block 716, the on-board computer 114 may compare the determinedcharacteristics against user profile data and/or breath analysis data(which may be stored in the data storage 228 or database 146) todetermine whether the user is inebriated. For example, statisticallysignificant difference in observed IR sensor data (such as intensity orwavelength) from expected normal IR sensor data when the vehicleoperator 106 breathes may be indicative of inebriation. If no baselineuser characteristics exist in a user profile for an occupant (such as anunknown occupant), a generic user profile may be used as a baseline. Insome embodiments, the on-board computer 114 may determine whetherobserved sensor data follows a pattern associated with inebriation,without reference to user-specific profile information.

At block 718, the on-board computer 114 may determine whether anabnormal condition associated with vehicle operator inebriation has beendetermined to exist. If no such abnormal situation is found, the method700 may terminate. If such an abnormal situation is found, the on-boardcomputer may determine an appropriate response at block 720. Theappropriate response may include preventing the vehicle 108 fromstarting if the vehicle operator 106 is determined to be inebriated. Ifinebriation is detected while the vehicle 108 is in operation, theon-board computer 114 may determine to take control of the vehicle 108in order to safely stop and shut down at an appropriate location —suchas shift an operating mode of an autonomous or semi-autonomous vehicleinto fully autonomous mode.

In some embodiments, the appropriate response may be determined basedupon the determined level of inebriation, environmental conditions(e.g., weather, traffic, location, etc.), other determined occupantcharacteristics, and/or the presence or absence of other occupantswithin the vehicle 108. For example, if no other vehicle occupants or noother authorized operators of the vehicle are identified, the on-boardcomputer 114 may prevent the vehicle 108 from starting and additionallyinitiate an electronic contact with an emergency contact stored in theuser's profile. If the on-board computer 114 determines one or moreadditional authorized vehicle operators identifies as occupants of thevehicle 108, however, the on-board computer 114 may determine anappropriate response includes presenting an alert indicating thatanother authorized vehicle operator should operate the vehicle 108. Insome embodiments, the on-board computer 114 may determine to contactappropriate family members or law enforcement agencies.

Additionally, or alternatively, the on-board computer 114 may provide anindication or alert to the occupants, such as by presenting an alertusing the speaker 122 or 246, display 202, or feedback device 128, whichmay provide a haptic alert to the vehicle operator 106 or otheroccupant. Once one or more appropriate responses have been determined,the on-board computer 114 may implement the determined responses atblock 722 using the speaker 122 or 246, display 202, feedback device128, communication unit 220, or other components of the system 100. Themethod 700 may then terminate.

Exemplary Ergonomic Monitoring & Response

FIG. 8 illustrates a flow diagram of an exemplary ergonomic monitoringand response method 800 that may be implemented by the vehicle occupantmonitoring system 100. The method 800 may monitor vehicle occupants todetermine when an ergonomic problem occurs and respond appropriately.

At block 812, the on-board computer 114 may collect sensor data from oneor more sensors within the vehicle 108. Data from IR sensors 120 and/orcameras 124 may be particularly relevant for determining usercharacteristics associated with occupant ergonomics.

At block 814, the on-board computer 114 may determine one or more useror occupant characteristics associated ergonomics, particularly skeletaland/or muscular characteristics. User characteristics such as posture,facing direction, movement, muscle exertion, muscle tension, and/ormuscle flushness may be particularly relevant, though othercharacteristics may likewise be determined.

At block 816, the on-board computer 114 may compare the determinedcharacteristics against user profile data to determine whether anergonomic problem exists. Because users may consistently repeatergonomic problems, however, the on-board computer 114 may also analyzethe user characteristics without reference to the user profile. Forexample, frequent shifting of weight or position adjustment movements bythe user may indicate an ergonomic problem. Similarly, localized muscletension or flushness may indicate that a problematic posture may beplacing strain on those portions of the user's musculature.

At block 818, the on-board computer 114 may determine whether anabnormal condition associated with user ergonomics has been determinedto exist. If no such abnormal situation is found, the method 800 mayterminate. If such an abnormal situation is found, the on-board computermay determine an appropriate response at block 820. The determinedresponse may include an alert or instructions to be presented to thevehicle operator 106 to improve the operator's posture. In someembodiments, the appropriate response may be determined based upon atleast in part upon the user profile. For example, users may selectwhether to receive ergonomic recommendations or reminders. Some usersmay welcome such recommendations, while others may prefer not to receivesuch recommendations.

The on-board computer 114 may further take account of other usercharacteristics or the vehicle environment in determining an appropriateresponse. For example, recommendations regarding posture may be delayedor suppressed if the sensor data indicate that the vehicle operator 106is agitated or anxious, or if the vehicle is operating in heavy trafficor adverse weather conditions.

Once one or more appropriate responses have been determined, theon-board computer 114 may implement the determined responses at block822 using the speaker 122 or 246, display 202, feedback device 128,communication unit 220, or other components of the system 100. Themethod 800 may then terminate.

Exemplary Accident Monitoring & Response

FIG. 9 illustrates a flow diagram of an exemplary accident monitoringand response method 900 that may be implemented by the vehicle occupantmonitoring system 100. The method 900 may monitor vehicle occupants todetermine when a vehicle accident has occurred based upon the effect onvehicle occupants and respond appropriately.

At block 912, the on-board computer 114 may collect sensor data from oneor more sensors within the vehicle 108. Data from IR sensors 120,cameras 124, and/or microphones 126 may be particularly relevant fordetermining user characteristics associated with vehicle accidents, butother sensor data related to the vehicle 108 may also be collected. Forexample, accelerometer data, GPS data, or data from vehicle systems(e.g., airbag systems, ABS systems, engine monitoring systems, etc.) maybe collected to determine vehicle accidents based upon the operation oroperating state of the vehicle 108.

At block 914, the on-board computer 114 may determine one or more useror occupant characteristics associated with vehicle accidents. Forexample, user heart rate, movement, or skeletal characteristics may beparticularly relevant to determination of the occurrence of a vehicleaccident. For example, sudden and simultaneous movements of multiplevehicle occupants in the same direction may indicate a sudden shift inmomentum of the entire vehicle 108, as typically occurs during acollision. Similarly, movement of vehicle occupant skeletal systems as awhole may be differentiated from movement of individual skeletal systemsegments (e.g., head, arms, hands, etc.). Movement of the entireskeletal system of an occupant may be determined as the introduction ofan external force, such as occurs during impact with another vehicle orobject. Changes in user characteristics such as heart rate or breathingrate may be particularly relevant following an accident, where sharpchanges may be indicative of accident or injury.

At block 916, the on-board computer 114 may determine whether a vehicleaccident has occurred based upon the user characteristics and/or vehiclesensor data. For example, the on-board computer 114 may determine theoccurrence of an accident based upon an abnormal user heart ratecombined with full-body movement and a sudden shift in accelerometerdata. If no baseline user characteristics exist in a user profile for anoccupant (such as an unknown occupant), a generic user profile may beused as a baseline. In such instances changes in occupantcharacteristics may be used to determine the occurrence of a vehicleaccident.

At block 918, the on-board computer 114 may determine whether anabnormal condition associated with a vehicle accident has beendetermined to exist. If no such abnormal situation is found, the method900 may terminate. If such an abnormal situation is found, the on-boardcomputer may determine an appropriate response at block 920. Theappropriate response may be determined based upon the received sensordata, determined occupant characteristics, the presence or absence ofother occupants within the vehicle 108, and/or the nature and severityof the accident. For example, an appropriate response to a minorcollision may include monitoring the vehicle occupant characteristicsfor indications of injuries and/or transmitting information regardingthe collision to an insurer for claims processing. In contrast, anappropriate response to a severe accident may include generating andtransmitting an automatic communication to an emergency responseservice.

In some embodiments, user characteristic data may be made accessible toemergency responders via the network 130. In further embodiments, theon-board computer 114 may determine that an appropriate responseincludes compiling sensor data and/or user characteristic dataassociated with a time period encompassing the vehicle accident, whichcompiled data may be stored in the memory 228 and/or transmitted to theserver 140 for storage and/or analysis. In this way, a record of thevehicle accident (and the vehicle occupants) may be preserved for lateruse in insurance claims processing, vehicle operation assessment,criminal investigations, or legal disputes. In some embodiments, theon-board computer 114 may prevent the vehicle 108 from restarting orcontinuing to operate following an accident until the vehicle operatorhas been determined to be in a suitable physical and mental condition tooperate the vehicle.

Once one or more appropriate responses have been determined, theon-board computer 114 may implement the determined responses at block922 using the speaker 122 or 246, display 202, feedback device 128,communication unit 220, or other components of the system 100. Themethod 900 may then terminate.

Exemplary Unauthorized Occupant Monitoring & Response

FIG. 10 illustrates a flow diagram of an exemplary unauthorized occupantmonitoring and response method 1000 that may be implemented by thevehicle occupant monitoring system 100. The method 1000 may monitorvehicle occupants to determine when one or more unauthorized persons arein the vehicle and respond appropriately. Unauthorized persons mayinclude identified occupants specifically prohibited from operating ortraveling in the vehicle 108, as well as unidentified occupants. Thepresence of unauthorized vehicle occupants may be monitored to preventtheft (of the vehicle or items within the vehicle), to allow the vehicleowner control over vehicle usage (e.g., parents may prevent teenagedrivers from driving multiple friends in a car), or to identifypotential security threats (e.g., a carjacking).

At block 1012, the on-board computer 114 may collect sensor data fromone or more sensors within the vehicle 108. Data from IR sensors 120and/or cameras 124 may be particularly relevant for determiningidentifying user characteristics.

At block 1014, the on-board computer 114 may determine one or more useror occupant characteristics that may be used to identify users oroccupants, such as facial features and heartbeat. In some embodiments,this may include determining behavioral characteristics of vehicleoccupants that may indicate unauthorized occupants status or purpose forbeing in the vehicle, such as biometric indicators of nervousness oragitation (e.g., elevated heart rate, sweating, etc.) or quick movements(e.g., constantly scanning the environment by quick turns of the head).

At block 1016, the on-board computer 114 may determine the whether oneor more vehicle occupants is an unauthorized occupant based upon theuser characteristics. In some embodiments, this may include determiningthe identities of vehicle occupants. In further embodiments, this mayinclude determining whether behavioral characteristics of an occupantindicate the occupant may be unauthorized, particularly in instanceswhere identification of the occupant is unavailable or uncertain.

Further embodiments may include determining whether occupants areunauthorized based in part upon a user profile or a vehicle profile. Forexample, a vehicle profile may be stored in the data storage 228 thatindicates a list of authorized vehicle operators, in which case anunauthorized occupant may be determined to be present if an occupant noton the list of authorized vehicle operators attempts to operate thevehicle 108. As another example, a user profile may indicate that onlyone unidentified occupant (or occupant not on a list of authorizedpassengers) may be a passenger in the vehicle at any given time.

At block 1018, the on-board computer 114 may determine whether anabnormal condition associated with an unauthorized vehicle occupant hasbeen determined to exist. If no such abnormal situation is found, themethod 1000 may terminate. If such an abnormal situation is found, theon-board computer may determine an appropriate response at block 1020.The appropriate response may be determined based upon the determinedoccupant characteristics, the presence or absence of authorized vehicleoccupants, the authorization level of any authorized vehicle occupantspresent, user profiles, and/or a vehicle profile associated with thevehicle 108.

For example, an authorized vehicle operator may be prompted to authorizethe unauthorized vehicle occupant (for a temporary or ongoing period) ifthe user profile allows such authorized vehicle operator to take suchaction. As another example, the on-board computer 114 may determine thatthe vehicle 108 should not start when no authorized operators areidentified as being present.

The on-board computer 114 may further determined whether an alert shouldbe presented, an alarm should sound, a vehicle owner should be notified,and/or police should be notified. In further embodiments, the on-boardcomputer 114 may determine that an appropriate response includescompiling sensor data and/or user characteristic data associated witheach unauthorized occupant, such as photographs of unauthorizedoccupants, which compiled data may be stored in the memory 228 and/ortransmitted to the server 140 for storage and/or analysis. In this way,a record of the unauthorized occupants may be preserved for later use inlater criminal investigations or for other purposes.

Once one or more appropriate responses have been determined, theon-board computer 114 may implement the determined responses at block1022 using the speaker 122 or 246, display 202, feedback device 128,communication unit 220, control connections to the vehicle engine orengine cut-off, or other components of the system 100. The method 1000may then terminate.

Exemplary Security Threat Monitoring & Response

FIG. 11 illustrates a flow diagram of an exemplary security threatmonitoring and response method 1100 that may be implemented by thevehicle occupant monitoring system 100. The method 1100 may monitorvehicle occupants to determine when a security threat exists and respondappropriately. In a manner similar to that of method 1000, the method1100 may distinguish between authorized occupants and unauthorizedoccupants within the vehicle 108. Additionally, the method 1100 maydistinguish between levels of authorization in some embodiments. Forexample, an authorized occupant may nonetheless be determined to be asecurity threat if sensor data indicate the presence of a weapon on theauthorized occupant's person.

At block 1112, the on-board computer 114 may collect sensor data fromone or more sensors within the vehicle 108. Data from IR sensors 120and/or cameras 124 may be particularly relevant for determining usercharacteristics associated with security threats. For example, IR sensordata may be used to identify concealed weapons based upon a temperaturedifferential compared with the occupant's body.

At block 1114, the on-board computer 114 may determine one or more useror occupant characteristics that may be used to identify securitythreats. This may include occupant identity. In some embodiments, thismay include determining behavioral characteristics of vehicle occupants,such as biometric indicators of nervousness or agitation (e.g., elevatedheart rate, sweating, etc.) or quick movements (e.g., constantlyscanning the environment by quick turns of the head).

At block 1116, the on-board computer 114 may determine whether asecurity threat exists (or is likely to exist) based upon the usercharacteristics. In some embodiments, this may include determining theidentities of vehicle occupants. In further embodiments, this mayinclude determining whether behavioral characteristics of an occupantindicate the occupant may be unstable, agitated, or aggressive. Furtherembodiments may include determining whether a weapon may be identified,and whether such weapon is concealed or exposed. If the weapon isexposed, the on-board computer 114 may further determine whether theweapon has been directed toward another vehicle occupant or otherwisebrandished. For example, the on-board computer 114 may determine thepresence of a concealed handgun based upon the difference in IR sensordata between the handgun and the occupant's body. In some embodiments,the on-board computer 114 may further determine whether such concealedweapon is on or about an authorized vehicle operator or occupant, andmay further determine whether such vehicle operator or occupant isauthorized to carry a weapon within the vehicle 108 based upon a userprofile.

At block 1118, the on-board computer 114 may determine whether anabnormal condition associated with a security threat has been determinedto exist. If no such abnormal situation is found, the method 1100 mayterminate. If such an abnormal situation is found, the on-board computermay determine an appropriate response at block 1120. The appropriateresponse may be determined based upon the determined occupantcharacteristics, the presence or absence of authorized vehicleoccupants, the authorization level of any authorized vehicle occupantspresent, user profiles, and/or a vehicle profile associated with thevehicle 108.

The on-board computer 114 may further determine whether an alert shouldbe presented, an alarm should sound, a vehicle owner should be notified,and/or police should be notified. The on-board computer 114 maydetermine to alert vehicle occupants (openly or discretely, such as viathe mobile device 110 or a haptic alert using the feedback device 128).

In some situations, the on-board computer 114 may determine anappropriate response includes notifying police or other law enforcementand/or emergency agencies or services. Such notifications may includeautomated electronic text or telephonic messages. For example, theon-board computer 114 may determine to notify police via a prerecordedor automated telephonic voice message when an unknown occupant isdetermined to have a weapon within the vehicle 108, which message may becommunicated via the communication module 220 of the on-board computer114 and/or the mobile device 110. Thus, the on-board computer 114 maycause the mobile device 110 to make an outgoing telephone call to anemergency response switchboard.

In further embodiments, the on-board computer 114 may determine that anappropriate response includes compiling sensor data and/or usercharacteristic data associated with each unauthorized occupant, such asphotographs of occupants, which compiled data may be stored in thememory 228 and/or transmitted to the server 140 for storage and/oranalysis. In this way, a record of the unauthorized occupants may bepreserved for later use in later criminal investigations or for otherpurposes. In yet further embodiments, the on-board computer 114 maydetermine that an appropriate response includes stopping the vehicle108.

Once one or more appropriate responses have been determined, theon-board computer 114 may implement the determined responses at block1022 using the speaker 122 or 246, display 202, feedback device 128,communication unit 220, control connections to the vehicle engine orengine cut-off, or other components of the system 100. The method 1100may then terminate.

Exemplary Operator Performance Monitoring & Response

FIG. 12 illustrates a flow diagram of an exemplary operator performancemonitoring and response method 1200 that may be implemented by thevehicle occupant monitoring system 100. The method 1200 may monitor avehicle operator to determine the quality of vehicle operatorperformance in piloting the vehicle and respond appropriately.

At block 1212, the on-board computer 114 may collect sensor data fromone or more sensors within the vehicle 108. Data from IR sensors 120,cameras 124, and/or microphones 126 may be particularly relevant fordetermining user characteristics, but other sensor data related to thevehicle 108 may also be collected. For example, accelerometer data, GPSdata, or data from vehicle systems (e.g., proximity sensors, guidance orentertainment systems, active cruise control, etc.) may be collected todetermine information regarding vehicle operation.

At block 1214, the on-board computer 114 may determine one or more usercharacteristics associated with the quality of vehicle operatorperformance. For example, user head orientation, gaze focus, blink rate,hand position, posture, or similar characteristics may be determined foruse in further determining the level of attention or alertness of thevehicle operator. Other similar characteristics may be determinedrelating to vehicle operator physical or emotional state, such asillness, drowsiness, distraction, or agitation.

At block 1216, the on-board computer 114 may determine a metric ofvehicle operator performance in operating the vehicle 108 based upon theuser characteristics and/or vehicle sensor data. For example, theon-board computer 114 may determine whether the vehicle operator 106 ischecking mirrors with sufficient frequency, whether the vehicle operator106 is looking away from the road too long, whether the vehicle operator106 is interacting with on-board guidance or entertainment systems toofrequently, or whether the vehicle operator 106 is interacting with amobile device 110 (such as a smartphone). In some embodiments, one ormore user performance metrics may be calculated (e.g., on a scale from 0to 100). In further embodiments, a total performance score may bedetermined for the vehicle operator 106 based upon the determined usercharacteristics and/or sensor data regarding the vehicle 108.

At block 1218, the on-board computer 114 may determine whether anabnormal condition associated with vehicle operator performance exists.An abnormal situation may be determined to exist if the vehicle operatorperformance metrics indicate high-risk vehicle operator or otherconditions outside the ordinary operating performance range of thevehicle operator and/or a target range for safe vehicle operation. If nosuch abnormal situation is found, the method 1200 may terminate.

In some embodiments, the method 1200 may further generate a reportregarding vehicle operator performance upon termination or periodically.Such report may include information relating to vehicle operatorperformance in one or more categories of vehicle operation (e.g.,attention to operation, excessive braking, etc.). In some embodiments,the report may be presented to an instructor or other interested partyas an indication of vehicle operator performance, such as during adriver education program.

If an abnormal situation is found, however, the on-board computer maydetermine an appropriate response at block 1220. The appropriateresponse may be determined based upon the received sensor data,determined occupant characteristics, one or more risk levels associatedwith the determined vehicle operator performance, and/or the userprofile associated with the vehicle operator. In some embodiments, theappropriate response may include an alert or message to the vehicleoperator 106, vehicle owner, vehicle insurer, or other interested party.

In further embodiments, the appropriate response may include generatinga report regarding the vehicle operator performance, such as describedabove. Such reports may be transmitted to an insurer for use indetermining or adjusting risk assessments, insurance rates, insurancepremiums, discounts, costs, terms, or other aspects of insurancepolicies associated with vehicle operators 106 and/or vehicles 108. Ininstances of extreme vehicle operator performance problems, the on-boardcomputer 114 may determine an appropriate response includes takingcontrol of the vehicle 108 or shutting down the vehicle 108.

Once one or more appropriate responses have been determined, theon-board computer 114 may implement the determined responses at block1222 using the speaker 122 or 246, display 202, feedback device 128,communication unit 220, or other components of the system 100. Themethod 1200 may then terminate.

Exemplary Risk Detection

FIG. 13 illustrates a flow diagram of an exemplary occupant riskdetection method 1300 that may be implemented by the vehicle occupantmonitoring system 100. The method 1300 may monitor the vehicle 108 whenno vehicle operator 106 is present to determine risks to vehicleoccupants and respond appropriately. Such vehicle occupants may includechildren or pets left unattended in a vehicle 108.

At block 1312, the on-board computer 114 may collect sensor data fromone or more sensors within the vehicle 108. Data from IR sensors 120and/or cameras 124 may be particularly relevant for determining usercharacteristics identifying and/or determining risks to vehicleoccupants.

At block 1314, the on-board computer 114 may determine one or morevehicle occupant characteristics associated with medical health ormedical emergencies. In some embodiments, this may include determiningwhether one or more occupants are within the vehicle 108. Occupantcharacteristics such as heart rate, pulse strength, heartbeat pattern,breathing rate, breathing volume, breathing pattern, facial features,vocal pattern, and/or skin temperature may be particularly relevant,though other characteristics may likewise be determined.

At block 1316, the on-board computer 114 may determine whether a risk toone or more occupants exists. For example, an increase in heart rate andskin temperature data may indicate that a child or pet is becomingoverheated, particularly if combined with sensor data indicating anincrease in temperature within the vehicle 108. As another example, avehicle occupant risk may be determined to exist if the vehicle occupanthas been left unattended for a period of time extending beyond anappropriate time threshold, which time threshold may vary based upon adetermined identity or category (e.g., pet, child, adult) of theoccupant. If no baseline user characteristics exist in a user profilefor an occupant, a generic user profile may be used as a baseline. Insuch instances changes in occupant characteristics may be used todetermine the existence of a vehicle occupant risk.

At block 1318, the on-board computer 114 may determine whether anabnormal condition associated with a vehicle occupant risk has beendetermined to exist. If no such abnormal situation is found, the method1300 may terminate. If such an abnormal situation is found, the on-boardcomputer may determine an appropriate response at block 1320.

The appropriate response may be determined based upon the receivedsensor data, determined occupant identity or category, determinedoccupant characteristics, the presence or absence of other occupantswithin the vehicle 108, and/or the nature and severity of the vehicleoperator risk. For example, an appropriate response to determining avehicle occupant is overheating may include opening windows of thevehicle (fully or partially) or turning on an air conditioning featureof the vehicle. In some embodiments, the on-board computer 114 maydetermine that an appropriate response includes presenting an alert to avehicle operator 106 or owner, which may be presented using the mobiledevice 110 or other means. In further embodiments, the on-board computer114 may cause an electronic text and/or telephonic message to betransmitted to an emergency service or agency regarding the vehicleoccupant risk via the network 130 using the communication module 220.

Once one or more appropriate responses have been determined, theon-board computer 114 may implement the determined responses at block1322 using the speaker 122 or 246, display 202, feedback device 128,communication unit 220, or other components of the system 100. Themethod 1300 may then terminate.

Exemplary Mobile Device Blocking

FIG. 14 illustrates a flow diagram of an exemplary mobile devicedisablement method 1400 that may be implemented by the vehicle occupantmonitoring system 100. The method 1400 may operate in conjunction withor in a manner similar to the methods 300-1300 described above. At block1402, the method may begin with receiving a command to begin monitoringthe occupants of the vehicle 108. Sensor data may be collected at block1404 and compared with profiles accessed at block 1406 to identify thevehicle operator 106 at block 1408. Once the vehicle operator 106 isidentified, one or more mobile devices 110 associated with the vehicleoperator 106 may be identified at block 1410. The on-board computer 114may then disable and/or block incoming communication into the identifiedmobile device 110 to reduce distractions and promote safe operation ofthe vehicle 108 at block 1412. Although the method 1400 is describedbelow as being implemented using the on-board computer 114, some or allof the steps may likewise be implemented using the mobile device 110,the server 140, or a combination of some or all of these.

At block 1402, the on-board computer 114 may receive a command tomonitoring occupants within the vehicle 108. The command may be enteredby the user, in some embodiments, or the command may be automaticallygenerated by the on-board computer 114. For example, the on-boardcomputer 114 may automatically implement the method 1400 upondetermining the presence of an occupant in the vehicle 108 or when thevehicle is started. The method 1400 may continue to be implemented whilethe vehicle remains in operation.

At block 1404, the on-board computer 114 may collect sensor data fromthe sensors within the vehicle. In particular, sensor data may becollected from one or more IR sensors 120, cameras 124, and/ormicrophones 126. The sensor data may be collected for a short period oftime, which may be taken as a snapshot of the vehicle occupants. Basedupon the sensor data, the on-board computer 114 may determine a numberof occupants of the vehicle, including one or more vehicle occupantspositioned within the vehicle 108 to operate the vehicle 108. In someembodiments, the on-board computer 114 may process part or all of thereceived sensor data to determine occupant characteristics forcomparison against occupant characteristics stored in user profiles, asdiscussed above. In further embodiments, occupant characteristics may bedetermined only for the one or more vehicle operators 106.

At block 1406, the on-board computer 114 may access one or more userprofiles stored in the data storage 228 or the database 146. The userprofiles may be selected from a set of user profiles associated with thevehicle 108. Additionally, or alternatively, the user profiles may besearched based upon the sensor data collected at block 1404 to findmatches for the vehicle operator 106.

At block 1408, the on-board computer 114 may identify one or morevehicle operators 106 based upon the user profiles. This may includeregressing the sensor data or derived occupant characteristics againstdata stored in a plurality of user profiles to determine a probabilityof a match between the vehicle operator 106 and one or more userprofiles. If no match can be found in the accessed user profiles, theon-board computer 114 may attempt to find a match with additional userprofiles stored in the system memory 228 or the database 146.Additionally, or alternatively, the on-board computer 114 may collectfurther sensor data and attempt to determine the identities of the oneor more vehicle operators 106 using the new sensor data. If the identityof one or more of the vehicle occupants cannot be determined withsufficient certainty, then the on-board computer 114 may identify suchoccupants as unknown users. In such instances where the vehicle operator106 is an unknown user, the on-board computer 114 may request orautomatically obtain additional data regarding the vehicle operator 106and generate a user profile for the user. Alternatively, the method 1400may terminate if the vehicle operator 106 cannot be identified.

At block 1410, the on-board computer 114 may identify one or more mobiledevices 110 associated with the one or more vehicle operators 106 basedupon the user profiles. For example, the user profile of the vehicleoperator 106 may indicate a smartphone and a wearable computing deviceassociated with the vehicle operator 106. In some embodiments, this mayinclude identifying one or more mobile devices 110 communicativelyconnected to the on-board computer 114, such as by a Bluetoothcommunication link. As there may be multiple mobile devices 110associated with vehicle occupants within the vehicle, the on-boardcomputer 114 may distinguish between mobile devices 110 associated withthe vehicle operator 106 (use of which should be limited during vehicleoperation) and mobile devices 110 associated with vehicle passengers(use of which may be left unrestricted).

At block 1412, the on-board computer 114 may disable or block incomingcommunications from reaching the one or more identified mobile devices110 associated with the vehicle operator 106. This may include sending asignal to the mobile device HO to temporarily disable communication orother features of the mobile device 110 while the vehicle is operating.In some embodiments, this may include preventing income and outgoingcommunication from the identified mobile devices 110. In alternativeembodiments, some types of communications may be permitted, while othertypes communications may be entirely disabled. For example, SMS textmessaging and e-mail may be disabled, while telephonic communicationsmay be allowed (or may be allowed only if operated through an interfaceof the on-board computer 114, rather than through the display 202 of themobile device 110). Where the mobile device 110 is paired with theon-board computer 114 (e.g., via Bluetooth or other wirelesscommunication connection), the on-board computer 114 may blockcommunication by presenting an automatic message to the mobile device110 in response to attempts to communicate using the on-board computer114. For example, an automatic text or prerecorded message may be sentto the mobile device 110 in response to an incoming communication.

Once the on-board computer 114 determines that the vehicle operator 106is no longer operating the vehicle 108, the on-board computer 114 maycause the mobile device 110 to resume normal or full functioning, andthe method 1400 may terminate. In some embodiments, the mobile device110 may continue attempting to operate normally during vehicleoperation, so the mobile device 110 may automatically resume normaloperation when the disabling or blocking functionality of the on-boardcomputer 114 is removed upon termination of the method 1400.

Other Matters

Although the following text sets forth a detailed description ofnumerous different embodiments, it should be understood that the legalscope of the invention is defined by the words of the claims set forthat the end of this patent. The detailed description is to be construedas exemplary only and does not describe every possible embodiment, asdescribing every possible embodiment would be impractical, if notimpossible. One could implement numerous alternate embodiments, usingeither current technology or technology developed after the filing dateof this patent, which would still fall within the scope of the claims.

It should also be understood that, unless a term is expressly defined inthis patent using the sentence “As used herein, the term ‘______’ ishereby defined to mean . . . ” or a similar sentence, there is no intentto limit the meaning of that term, either expressly or by implication,beyond its plain or ordinary meaning, and such term should not beinterpreted to be limited in scope based on any statement made in anysection of this patent (other than the language of the claims). To theextent that any term recited in the claims at the end of this patent isreferred to in this patent in a manner consistent with a single meaning,that is done for sake of clarity only so as to not confuse the reader,and it is not intended that such claim term be limited, by implicationor otherwise, to that single meaning. Further, unless a claim element isdefined by expressly reciting the words “means for” or “step for” and afunction without the recital of any structure, it is not intended thatthe scope of any claim element be interpreted based on the applicationof 35 U.S.C. § 112(f).

As used herein, the term “vehicle” may refer to any of a number ofmotorized transportation devices. A vehicle may be a car, truck, bus,train, boat, plane, etc. Additionally, as used herein, the term“impairment” refers to any of a number of conditions that may reducevehicle operator performance. A vehicle operator may be impaired if thevehicle operator is drowsy, asleep, distracted, intoxicated, ill,injured, suffering from a sudden onset of a medical condition, or in animpaired emotional state such as anxiety, agitation, aggression,nervousness, hyperactivity, or mania.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Additionally, certain embodiments are described herein as includinglogic or a number of routines, subroutines, applications, orinstructions. These may constitute either software (code embodied on anon-transitory, tangible machine-readable medium) or hardware. Inhardware, the routines, etc., are tangible units capable of performingcertain operations and may be configured or arranged in a certainmanner. In example embodiments, one or more computer systems (e.g., astandalone, client or server computer system) or one or more hardwaremodules of a computer system (e.g., a processor or a group ofprocessors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a field programmable gatearray (FPGA) or an application-specific integrated circuit (ASIC) toperform certain operations. A hardware module may also compriseprogrammable logic or circuitry (e.g., as encompassed within ageneral-purpose processor or other programmable processor) that istemporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multipleof such hardware modules exist contemporaneously, communications may beachieved through signal transmission (e.g., over appropriate circuitsand buses) that connect the hardware modules. In embodiments in whichmultiple hardware modules are configured or instantiated at differenttimes, communications between such hardware modules may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware modules have access. Forexample, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) permanently configured toperform the relevant operations. Whether temporarily or permanently,configured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods or routines described herein may be at leastpartially processor-implemented. For example, at least some of theoperations of a method may be performed by one or more processors orprocessor-implemented hardware modules. The performance of certain ofthe operations may be distributed among the one or more processors, notonly residing within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but deployed across a number of machines. In some example embodiments,the one or more processors or processor-implemented modules may belocated in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or a combination thereof), registers, or othermachine components that receive, store, transmit, or displayinformation.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present). A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the description. Thisdescription, and the claims that follow, should be read to include oneor at least one and the singular also includes the plural unless it isobvious that it is meant otherwise.

This detailed description is to be construed as exemplary only and doesnot describe every possible embodiment, as describing every possibleembodiment would be impractical, if not impossible. One could implementnumerous alternate embodiments, using either current technology ortechnology developed after the filing date of this application.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs forsystem and a method for assigning mobile device data to a vehiclethrough the disclosed principles herein. Thus, while particularembodiments and applications have been illustrated and described, it isto be understood that the disclosed embodiments are not limited to theprecise construction and components disclosed herein. Variousmodifications, changes and variations, which will be apparent to thoseskilled in the art, may be made in the arrangement, operation anddetails of the method and apparatus disclosed herein without departingfrom the spirit and scope defined in the appended claims.

The particular features, structures, or characteristics of any specificembodiment may be combined in any suitable manner and in any suitablecombination with one or more other embodiments, including the use ofselected features without corresponding use of other features. Inaddition, many modifications may be made to adapt a particularapplication, situation or material to the essential scope and spirit ofthe present invention. It is to be understood that other variations andmodifications of the embodiments of the present invention described andillustrated herein are possible in light of the teachings herein and areto be considered part of the spirit and scope of the present invention.

1. A computer-implemented method for monitoring a vehicle occupant of avehicle, the method comprising, via one or more processors; receiving,from one or more imaging sensors disposed within the vehicle, imagingsensor data regarding the vehicle occupant; determining one or morevehicle occupant characteristics for the vehicle occupant based upon thereceived imaging sensor data, wherein the one or more vehicle occupantcharacteristics include (i) one or more skeletal characteristics of thevehicle occupant, including at least one joint and a plurality ofskeletal segments of at least one arm of the vehicle occupant, and (ii)one or more muscular characteristics of a muscle group associated withat least one of the skeletal segments of the vehicle occupant based atleast in part upon the plurality of skeletal segments, the one or moremuscular characteristics including at least one of temperature orflushness of the muscle group; determining whether an abnormal situationexists based upon the one or more determined vehicle occupantcharacteristics associated with the vehicle occupant, including the oneor more skeletal characteristics and the one or more muscularcharacteristics, wherein the abnormal situation relates to one or moreof the following types of abnormal situations: a medical emergency, ahealth risk, an accident risk, an impairment of the vehicle occupant, ora security threat; and causing one or more responses to the abnormalsituation to be implemented.
 2. The computer-implemented method of claim1, wherein the one or more skeletal characteristics indicate theposition of a plurality of segments of the vehicle occupant's body,wherein the plurality of segments include the vehicle occupant's headand at least a portion of a limb of the vehicle occupant.
 3. Thecomputer-implemented method of claim 1, further comprising: identifyingthe vehicle occupant by comparing the one or more determined vehicleoccupant characteristics with data regarding characteristics stored in auser profile.
 4. The computer-implemented method of claim 3, whereindetermining whether the abnormal situation exists includes determiningwhether the one or more determined vehicle occupant characteristics arebeyond a baseline range for the vehicle occupant based upon the dataregarding the characteristics stored in the user profile of the vehicleoccupant.
 5. The computer-implemented method of claim 1, wherein the oneor more responses include controlling vehicle operation by an on-boardcomputer system.
 6. The computer-implemented method of claim 1, whereinthe one or more responses include adjusting an environmental conditionwithin the vehicle.
 7. The computer-implemented method of claim 1,wherein the one or more responses include communicating a message to anemergency response service.
 8. The computer-implemented method of claim1, wherein the one or more responses include terminating vehicleoperation.
 9. The computer-implemented method of claim 1, wherein theone or more responses include communicating sensor data to one or morecomputing devices associated with emergency response personnel.
 10. Thecomputer-implemented method of claim 1, wherein the one or more imagingsensors include one or more infrared sensors disposed within thevehicle.
 11. A computer system for monitoring a vehicle occupant of avehicle, the computer system comprising: one or more processors; one ormore imaging sensors disposed within the vehicle and communicativelyconnected to the one or more processors; and a program memory coupled tothe one or more processors and storing executable instructions that whenexecuted by the one or more processors cause the computer system to:receive imaging sensor data regarding the vehicle occupant from the oneor more imaging sensors; determine one or more vehicle occupantcharacteristics for the vehicle occupant based upon the received imagingsensor data, wherein the one or more vehicle occupant characteristicsinclude (i) one or more skeletal characteristics of the vehicleoccupant, including at least one joint and a plurality of skeletalsegments of at least one arm of the vehicle occupant, and (ii) one ormore muscular characteristics of a muscle group associated with at leastone of the skeletal segments of the vehicle occupant based at least inpart upon the plurality of skeletal segments, the one or more muscularcharacteristics including at least one of temperature or flushness ofthe muscle group; determine whether an abnormal situation exists basedupon the one or more determined vehicle occupant characteristicsassociated with the vehicle occupant, including the one or more skeletalcharacteristics and the one or more muscular characteristics, whereinthe abnormal situation relates to one or more of the following types ofabnormal situations: a medical emergency, a health risk, an accidentrisk, an impairment of the vehicle occupant, or a security threat; andcause one or more responses to the abnormal situation to be implemented.12. The computer system of claim 11, wherein the one or more skeletalcharacteristics indicates the position of a plurality of segments of thevehicle occupant's body, wherein the plurality of segments include thevehicle occupant's head and at least a portion of a limb of the vehicleoccupant.
 13. The computer system of claim 11, wherein the programmemory further stores executable instructions that cause the computersystem to: identify the vehicle occupant based upon the one or moredetermined vehicle occupant characteristics by comparing the one or moredetermined vehicle occupant characteristics with data regardingcharacteristics stored in a user profile.
 14. The computer system ofclaim 13, wherein the executable instructions that cause the computersystem to determine whether the abnormal situation exists further causethe computer system to determine whether the one or more determinedvehicle occupant characteristics are beyond a baseline range for thevehicle occupant based upon the data regarding the characteristicsstored in the user profile of the vehicle occupant.
 15. The computersystem of claim 11, wherein the one or more responses includecontrolling vehicle operation by an on-board computer system.
 16. Thecomputer system of claim 11, wherein the one or more responses includeadjusting an environmental condition within the vehicle.
 17. Thecomputer system of claim 11, wherein the one or more responses includecommunicating a message to an emergency response service.
 18. Thecomputer system of claim 11, wherein the one or more responses includeterminating vehicle operation.
 19. The computer system of claim 11,wherein the one or more imaging sensors include one or more infraredsensors disposed within the vehicle.
 20. A tangible, non-transitorycomputer-readable medium storing instructions for monitoring a vehicleoccupant of a vehicle that, when executed by at least one processor of acomputer system, cause the computer system to: receive imaging sensordata regarding the vehicle occupant from one or more imaging sensorsdisposed within the vehicle; determine one or more vehicle occupantcharacteristics for the vehicle occupant based upon the received imagingsensor data, wherein the one or more vehicle occupant characteristicsinclude (i) one or more skeletal characteristics of the vehicleoccupant, including at least one joint and a plurality of skeletalsegments of at least one arm of the vehicle occupant, and (ii) one ormore muscular characteristics of a muscle group associated with at leastone of the skeletal segments of the vehicle occupant based at least inpart upon the plurality of skeletal segments, the one or more muscularcharacteristics including at least one of temperature or flushness ofthe muscle group; determine whether an abnormal situation exists basedupon the one or more determined vehicle occupant characteristicsassociated with the vehicle occupant, including the one or more skeletalcharacteristics and the one or more muscular characteristics, whereinthe abnormal situation relates to one or more of the following types ofabnormal situations: a medical emergency, a health risk, an accidentrisk, an impairment of the vehicle occupant, or a security threat; andcause one or more response to the abnormal situation to be implemented.